Resolve disagreements with a process, then turn a week of numbers into actions leadership can act on
Day 44 of 66
The audit surfaces disagreements; something has to resolve them, and something has to communicate what you learned. Today, on the job, you build both: an adjudication workflow (who resolves a disagreement, when, and how the resolution feeds the guideline) and a weekly quality report that turns raw numbers into recommended actions. These are the two artifacts that turn "I ran an audit" into "I run a quality operation."
Adjudication is the defined process for resolving a disagreement to a single correct answer — and, critically, for making that resolution improve the system. Without a defined process, disagreements pile up, get resolved by whoever shouts loudest, and the lesson is lost. A good workflow answers three questions explicitly.
A named adjudicator — usually the lead or a designated senior annotator with deep guideline knowledge. Not "the group decides," which dissolves accountability. One person owns the call, and owns writing down the reasoning.
On a clear rule, not a vibe. Common triggers: annotators split on a consensus item, a gold item was missed, or an annotator flags a case as ambiguous (the Judgment Ritual, Day 5). Severe or high-stakes cases escalate to senior review faster.
The adjudicator decides the correct label and classifies the root cause: guideline gap, gold error, or annotator miscalibration. The resolution then updates the guideline (a new worked example or rule), fixes the gold, or schedules a calibration note. Disagreement is data: every adjudicated case is a guideline patch waiting to happen.
An adjudication that ends at "the answer is accept" is half-done. An adjudication that ends at "the answer is accept, and we added Example 7 to the guideline so this never splits the team again" is the full loop. The resolution must always name where the fix lands. If it doesn't change a guideline, a gold set, or a person, you'll adjudicate the same case again next week.
The report is how the operation becomes visible to your team and to leadership. The failure mode is a wall of numbers nobody acts on. The fix is a fixed template where every number is paired with an interpretation and a recommended action.
"Accuracy fell to 83%" is a fact. "Accuracy fell to 83% — driven by a new prompt category the guideline doesn't cover; I've drafted Rule 12 and will recalibrate the two annotators who missed it by Friday" is a report. The first invites a meeting. The second closes it. Every issue in your report must end in a verb and an owner.
Numbers without a story leave the reader to invent one — usually a worse one ("is the team slacking?"). The narrative section is where you, who actually ran the loop, explain causation. It's the difference between a report that builds trust and one that triggers panic.
| Numbers alone | Numbers + narrative |
|---|---|
| Accuracy: 92% → 84% | Accuracy dropped 8 pts because we onboarded 3 annotators mid-week; their work is now isolated and in calibration. Veteran accuracy held at 93%. |
| Agreement: κ 0.61 → 0.48 | Agreement fell after a guideline "clarification" that introduced a new ambiguity on borderline aesthetics; reverting it, re-measuring next week. |
| Top issue: text-render fails | Text-rendering failures spiked — traced to the new image model, not the labels. Flagged upstream to the model team; guideline unchanged. |
Sometimes the right action is not a data fix at all — it's reporting upward that the model changed. A lead who can tell "our labels drifted" from "the model's outputs changed" is worth their weight. The narrative is where you make that distinction explicit so the right team owns the problem.
qa_audit.py output (or invent a plausible week) and fill in the weekly report template — every number paired with an interpretation and an action.An annotator resolves a disputed item and moves on. A lead makes every resolution name where the fix lands — a new worked example, a corrected gold item, a calibration note — and translates a week of numbers into a report that ends each issue in a verb and an owner, so leadership reads an action plan, not a wall of stats.
Say this in an interview: "My adjudication never ends at 'the answer is accept' — it ends at 'and here's the guideline example that stops this from splitting the team again.' And in my weekly report I separate 'our labels drifted' from 'the model's outputs changed,' so the right team owns the problem."